{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ERNIE SFT Best Practices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 Introduction to SFT\n",
    "### 1.1 What is SFT?\n",
    "SFT (Supervised Fine-Tuning) is a key technique for customizing pre-trained large models. By continuing to train the model with a **small amount of high-quality labeled data**, it's possible to enhance its performance in specific domains at a low cost and with high speed.\n",
    "\n",
    "### 1.2 When to Use SFT\n",
    "- ✅ Applicable Scenarios:\n",
    "  - Task Adaptation: Strengthening the model's understanding of specific tasks (e.g., text classification, information extraction).\n",
    "  - Reinforcing existing abilities or knowledge the model already possesses.\n",
    "  - Teaching the model to handle more complex, composite tasks (provided it already has the ability to complete the relevant sub-tasks).\n",
    "\n",
    "- ❌ Inapplicable Scenarios:\n",
    "  - Injecting new internal knowledge into the model.\n",
    "  - Forcing the model to learn abilities it doesn't have.\n",
    "  - Completely correcting inherent biases from the pre-trained model.\n",
    "  - Tackling scenarios that require complex reasoning or long-term memory (which may require techniques like reinforcement learning).\n",
    "\n",
    "\n",
    "### 1.3 General Steps for SFT\n",
    "1. Prepare evaluation data and metrics, test the base model's capabilities, and analyze its shortcomings and issues.\n",
    "2. Conduct comprehensive prompt engineering.\n",
    "3. Determine if SFT is the right approach; prepare the training data.\n",
    "4. Training. Design training parameters based on the task's characteristics and the data. Key hyperparameters to focus on include:\n",
    "    - Batch size\n",
    "    - Number of training epochs / Maximum training steps\n",
    "    - Learning rate scheduler type / Initial learning rate\n",
    "    - Warmup steps\n",
    "5. Performance Evaluation. Using the same evaluation data and metrics, assess the performance of the fine-tuned model. If it falls short of expectations, consider further optimizing the training data or tuning the hyperparameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Introduction to the Best Practice Task\n",
    "\n",
    "### 2.1 Task Description\n",
    "For this best practice example, we've chosen a function-calling task. Given a set of tool functions and a multi-turn dialogue, the model's objective is to select the most appropriate tool to complete the task based on the dialogue context. For simplicity, the model only needs to output the name of the tool function.\n",
    "\n",
    "### 2.2 Glossary\n",
    "To make things clearer, we'll define the following terms:\n",
    "- **session**: A multi-turn dialogue, consisting of multiple question-and-answer pairs.\n",
    "- **round**: A single dialogue turn, including one question and one answer.\n",
    "- **history**: Session data formatted according to the OpenAI API standard.\n",
    "\n",
    "### 2.3 Data Source\n",
    "We use the popular open-source `glaive_toolcall` dataset ([link](https://huggingface.co/datasets/llamafactory/glaive_toolcall_en)). This dataset contains 1,000 entries each in Chinese and English, formatted in the ShareGPT style with the following fields:\n",
    "- `conversations`: The dialogue context, in `[{\"from\": \"\", \"value\": \"\"}]` format. The `from` field includes:\n",
    "  - `human`: User input.\n",
    "  - `gpt`: Model output (here, `gpt` is used generically to refer to a large model conversational assistant).\n",
    "  - `function_call`: The specific tool being called and its parameters.\n",
    "  - `observation`: The result of the tool call.\n",
    "- `tools`: The list of available tools and their descriptions.\n",
    "\n",
    "### 2.4 Evaluation Metrics\n",
    "This is a multi-class classification task. We will evaluate the model's performance using accuracy, macro-average, micro-average, and weighted-average metrics.\n",
    "\n",
    "### 2.5 Experiment Environment\n",
    "This tutorial uses the following environment:\n",
    "- 1 x 80GB A800 GPU\n",
    "- CUDA Version: 12.3\n",
    "- CUDA Driver: 525.125.06\n",
    "- nvcc: 12.3\n",
    "- gcc: 12.2\n",
    "- Python Version: 3.10.12\n",
    "\n",
    "### 2.6 Dependencies\n",
    "- **ERNIEKit**: The ERNIE large model toolchain, which provides an end-to-end workflow for training, compressing, and deploying the ERNIE 4.5 series models. It is based on PaddlePaddle v3.1 and supports training on several mainstream domestic chips.\n",
    "- **ERNIEKit WebUI**: A graphical interface that supports training, interactive dialogue, performance evaluation, model exporting, and other features. [Docs](docs/cli_webui_usage.md)\n",
    "- **[Optional] visualdl**: A tool for visualizing information like loss curves, which is already included in ERNIEKit.\n",
    "- **ERNIEKit inference scripts**.\n",
    "- **The Python dependencies** used in this tutorial.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The Python dependencies\n",
    "import json\n",
    "import random\n",
    "from collections import defaultdict\n",
    "from string import Template\n",
    "\n",
    "import pandas as pd\n",
    "from pyecharts import options as opts\n",
    "from pyecharts.charts import Bar, Line\n",
    "from pyecharts.globals import ThemeType\n",
    "\n",
    "RANDOM_SEED = 2025"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Data Preprocessing\n",
    "\n",
    "First, let's examine the `glaive_toolcall` dataset. We can see that there are cases where multiple function calls occur within the same session. For such data, we will split them into multiple sessions, using each function call as a boundary.\n",
    "\n",
    "Next, we will process the `glaive_toolcall` dataset to align with the data formats required by ERNIEKit for training and inference.\n",
    "  - **Training data**: A `jsonl` file, with each line as `{\"system\": \"\", \"src\": [], \"tgt\": [], \"label\": []}`\n",
    "    - `system` (str, optional): A prompt that sets system information, such as persona, task instructions, style definitions, etc.\n",
    "    - `src` (list(str)): The user's multi-turn dialogue content, arranged by turn and corresponding to `tgt`.\n",
    "    - `tgt` (list(str)): The system's multi-turn responses, arranged by turn and corresponding to `src`.\n",
    "    - `label` (list(int), optional): Marks whether the current dialogue turn should be used for training. `1` indicates it's included in training (for loss calculation), while `0` means it's excluded. If this field is omitted, all turns are used for training.\n",
    "  - **Inference data**: A `jsonl` file, with each line as `[{\"role\": \"\", \"content\": \"\"}]`. The `role` field can be `system`, `user`, or `assistant`.\n",
    "\n",
    "Considering the formats above, we will reformat the `glaive_toolcall` dataset into the following structure: a `jsonl` file where each line is `[{\"role\": \"\", \"content\": \"\"}]`. The `role` can be `system`, `user`, or `assistant`. The `user` and `assistant` roles appear in pairs according to the dialogue sequence, and the final `assistant` message records the name of the tool called."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Prepare the Raw Data\n",
    "First, let's download the open-source dataset. Save at `cookbook/data/glaive_toolcall_en_1k.json`。\n",
    "\n",
    "![sft_download_dataset](../assets/sft_download_dataset_en.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To facilitate processing, we will convert the ShareGPT format to the OpenAI API format and log the tool call information. We have defined several functions for data preprocessing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_function(tools, func_name_with_desc: dict):\n",
    "    \"\"\"Parses the tool list to get tool names and descriptions.\n",
    "\n",
    "    Args:\n",
    "        tools: The list of tools.\n",
    "        func_name_with_desc: A dictionary to store the mapping of tool names to their descriptions.\n",
    "\n",
    "    Returns:\n",
    "        func_name_with_desc: The updated dictionary mapping tool names to descriptions.\n",
    "    \"\"\"\n",
    "    for tool in tools:\n",
    "        tool_name = tool.get(\"name\", \"\")\n",
    "        tool_desc = tool.get(\"description\", \"\")\n",
    "        func_name_with_desc[tool_name] = tool_desc\n",
    "    return func_name_with_desc\n",
    "\n",
    "\n",
    "def is_valid_history(history):\n",
    "    \"\"\"Checks if the conversation history is valid: user and assistant roles must appear in pairs, and the last turn must be from the assistant.\n",
    "\n",
    "    Args:\n",
    "        history: Multi-turn dialogue data in OpenAI API format.\n",
    "\n",
    "    Returns:\n",
    "        bool: True if the history is valid, False otherwise.\n",
    "    \"\"\"\n",
    "    # Remove the system prompt for easier calculation.\n",
    "    history = [h for h in history if h.get(\"role\", \"\") != \"system\"]\n",
    "    if len(history) < 2 or len(history) % 2 != 0:\n",
    "        return False\n",
    "    for i in range(1, len(history), 2):\n",
    "        if history[i][\"role\"] != \"assistant\" or history[i - 1][\"role\"] != \"user\":\n",
    "            return False\n",
    "    else:\n",
    "        return True\n",
    "\n",
    "\n",
    "def conversations_to_history(conversations):\n",
    "    \"\"\"Converts conversation data from ShareGPT format to OpenAI API format.\n",
    "\n",
    "    Args:\n",
    "        conversations: Conversation data in ShareGPT format.\n",
    "\n",
    "    Yields:\n",
    "        history: Multi-turn dialogue data in OpenAI API format.\n",
    "    \"\"\"\n",
    "    NO_FUNCTION_CALL_MARK = \"<|NO_FUNCTION_CALL|>\"\n",
    "    had_function_call = False\n",
    "    history = []\n",
    "    for conversation in conversations:\n",
    "        if conversation.get(\"from\", \"\") == \"human\":\n",
    "            history.append({\"role\": \"user\", \"content\": conversation.get(\"value\", \"\")})\n",
    "        elif conversation.get(\"from\", \"\") == \"function_call\":\n",
    "            history.append({\"role\": \"assistant\", \"content\": json.loads(conversation.get(\"value\", \"\")).get(\"name\", \"\")})\n",
    "            if is_valid_history(history):\n",
    "                yield history\n",
    "                had_function_call = True\n",
    "                history.pop()\n",
    "                # If there are more turns in the dialogue, the 'assistant' field will use the 'gpt' content\n",
    "                # from the original data (not the 'function_call').\n",
    "                # Therefore, we remove the current 'assistant' to prepare for the next turn.\n",
    "\n",
    "        elif conversation.get(\"from\", \"\") == \"gpt\":\n",
    "            if had_function_call:\n",
    "                history.append({\"role\": \"assistant\", \"content\": conversation.get(\"value\", \"\")})\n",
    "                had_function_call = False\n",
    "            else:\n",
    "                history.append({\"role\": \"assistant\", \"content\": NO_FUNCTION_CALL_MARK})\n",
    "                if is_valid_history(history):\n",
    "                    yield history\n",
    "                    history.pop()\n",
    "                    history.append({\"role\": \"assistant\", \"content\": conversation.get(\"value\", \"\")})\n",
    "        else:\n",
    "            pass\n",
    "\n",
    "\n",
    "def history_to_train(history, system_prompt: str = \"\"):\n",
    "    \"\"\"Converts multi-turn dialogue from OpenAI API format to ERNIE training data format.\n",
    "\n",
    "    Args:\n",
    "        history: Multi-turn dialogue data in OpenAI API format.\n",
    "\n",
    "    Returns:\n",
    "        data: Training data in ERNIE format.\n",
    "    \"\"\"\n",
    "    data = {\"system\": system_prompt, \"src\": [], \"tgt\": [], \"label\": []}\n",
    "    if not is_valid_history(history):\n",
    "        return {}\n",
    "\n",
    "    for h in history:\n",
    "        role = h.get(\"role\", \"\")\n",
    "        content = h.get(\"content\", \"\")\n",
    "        if role == \"user\":\n",
    "            data[\"src\"].append(content)\n",
    "        elif role == \"assistant\":\n",
    "            data[\"tgt\"].append(content)\n",
    "        data[\"label\"] = [0] * (len(data[\"tgt\"]) - 1) + [1]\n",
    "    return data\n",
    "\n",
    "\n",
    "def history_to_test(history, system_prompt: str = \"\"):\n",
    "    \"\"\"Converts multi-turn dialogue from OpenAI API format to ERNIE inference data format.\n",
    "\n",
    "    Args:\n",
    "        history: Multi-turn dialogue data in OpenAI API format.\n",
    "\n",
    "    Returns:\n",
    "        data: Inference data in ERNIE format.\n",
    "    \"\"\"\n",
    "    return [{\"role\": \"system\", \"content\": system_prompt}] + history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We call the functions defined above to process the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_data_path = \"../data/glaive_toolcall_en_1k.json\"\n",
    "history_path = raw_data_path.replace('.json', '_history.jsonl')\n",
    "func_name_with_desc = {}\n",
    "\n",
    "with open(raw_data_path, \"r\") as fin:\n",
    "    with open(history_path, \"w\") as fout:\n",
    "        data_list = json.load(fin)\n",
    "        for data in data_list:\n",
    "            for history in conversations_to_history(data.get(\"conversations\", [])):\n",
    "                fout.write(json.dumps(history, ensure_ascii=False) + \"\\n\")\n",
    "            # Record the function call information as well\n",
    "            parse_function(json.loads(data.get(\"tools\", [])), func_name_with_desc)\n",
    "\n",
    "func_name_with_desc[\"<|NO_FUNCTION_CALL|>\"] = \"No function call is needed\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Analyze the Raw Data\n",
    "Let's observe how each tool is used, and then construct the training and test sets accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "func_cnt = {}\n",
    "with open(history_path, 'r') as fin:\n",
    "    for line in fin:\n",
    "        data = json.loads(line)\n",
    "        func_name = data[-1].get(\"content\", \"\")\n",
    "        func_cnt[func_name] = func_cnt.get(func_name, 0) + 1\n",
    "\n",
    "func_cnt = dict(sorted(func_cnt.items(), key=lambda x: x[1], reverse=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's perform a simple visual analysis of the tool call distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<script>\n",
       "    require.config({\n",
       "        paths: {\n",
       "            'echarts':'https://assets.pyecharts.org/assets/v5/echarts.min'\n",
       "        }\n",
       "    });\n",
       "</script>\n",
       "\n",
       "        <div id=\"22c006e9322c4a138bd76ffec17ec39f\" style=\"width:800px; height:400px;\"></div>\n",
       "\n",
       "<script>\n",
       "        require(['echarts'], function(echarts) {\n",
       "                var chart_22c006e9322c4a138bd76ffec17ec39f = echarts.init(\n",
       "                    document.getElementById('22c006e9322c4a138bd76ffec17ec39f'), 'light', {renderer: 'canvas'});\n",
       "                var option_22c006e9322c4a138bd76ffec17ec39f = {\n",
       "    \"animation\": true,\n",
       "    \"animationThreshold\": 2000,\n",
       "    \"animationDuration\": 1000,\n",
       "    \"animationEasing\": \"cubicOut\",\n",
       "    \"animationDelay\": 0,\n",
       "    \"animationDurationUpdate\": 300,\n",
       "    \"animationEasingUpdate\": \"cubicOut\",\n",
       "    \"animationDelayUpdate\": 0,\n",
       "    \"aria\": {\n",
       "        \"enabled\": false\n",
       "    },\n",
       "    \"series\": [\n",
       "        {\n",
       "            \"type\": \"line\",\n",
       "            \"name\": \"Call Count\",\n",
       "            \"connectNulls\": false,\n",
       "            \"xAxisIndex\": 0,\n",
       "            \"symbol\": \"circle\",\n",
       "            \"symbolSize\": 4,\n",
       "            \"showSymbol\": true,\n",
       "            \"smooth\": true,\n",
       "            \"clip\": true,\n",
       "            \"step\": false,\n",
       "            \"stackStrategy\": \"samesign\",\n",
       "            \"data\": [\n",
       "                [\n",
       "                    \"1\",\n",
       "                    1678\n",
       "                ],\n",
       "                [\n",
       "                    \"2\",\n",
       "                    48\n",
       "                ],\n",
       "                [\n",
       "                    \"3\",\n",
       "                    44\n",
       "                ],\n",
       "                [\n",
       "                    \"4\",\n",
       "                    34\n",
       "                ],\n",
       "                [\n",
       "                    \"5\",\n",
       "                    33\n",
       "                ],\n",
       "                [\n",
       "                    \"6\",\n",
       "                    32\n",
       "                ],\n",
       "                [\n",
       "                    \"7\",\n",
       "                    31\n",
       "                ],\n",
       "                [\n",
       "                    \"8\",\n",
       "                    28\n",
       "                ],\n",
       "                [\n",
       "                    \"9\",\n",
       "                    28\n",
       "                ],\n",
       "                [\n",
       "                    \"10\",\n",
       "                    24\n",
       "                ],\n",
       "                [\n",
       "                    \"11\",\n",
       "                    22\n",
       "                ],\n",
       "                [\n",
       "                    \"12\",\n",
       "                    22\n",
       "                ],\n",
       "                [\n",
       "                    \"13\",\n",
       "                    19\n",
       "                ],\n",
       "                [\n",
       "                    \"14\",\n",
       "                    18\n",
       "                ],\n",
       "                [\n",
       "                    \"15\",\n",
       "                    17\n",
       "                ],\n",
       "                [\n",
       "                    \"16\",\n",
       "                    17\n",
       "                ],\n",
       "                [\n",
       "                    \"17\",\n",
       "                    16\n",
       "                ],\n",
       "                [\n",
       "                    \"18\",\n",
       "                    12\n",
       "                ],\n",
       "                [\n",
       "                    \"19\",\n",
       "                    12\n",
       "                ],\n",
       "                [\n",
       "                    \"20\",\n",
       "                    11\n",
       "                ],\n",
       "                [\n",
       "                    \"21\",\n",
       "                    10\n",
       "                ],\n",
       "                [\n",
       "                    \"22\",\n",
       "                    8\n",
       "                ],\n",
       "                [\n",
       "                    \"23\",\n",
       "                    8\n",
       "                ],\n",
       "                [\n",
       "                    \"24\",\n",
       "                    8\n",
       "                ],\n",
       "                [\n",
       "                    \"25\",\n",
       "                    7\n",
       "                ],\n",
       "                [\n",
       "                    \"26\",\n",
       "                    7\n",
       "                ],\n",
       "                [\n",
       "                    \"27\",\n",
       "                    7\n",
       "                ],\n",
       "                [\n",
       "                    \"28\",\n",
       "                    7\n",
       "                ],\n",
       "                [\n",
       "                    \"29\",\n",
       "                    7\n",
       "                ],\n",
       "                [\n",
       "                    \"30\",\n",
       "                    7\n",
       "                ],\n",
       "                [\n",
       "                    \"31\",\n",
       "                    6\n",
       "                ],\n",
       "                [\n",
       "                    \"32\",\n",
       "                    5\n",
       "                ],\n",
       "                [\n",
       "                    \"33\",\n",
       "                    5\n",
       "                ],\n",
       "                [\n",
       "                    \"34\",\n",
       "                    5\n",
       "                ],\n",
       "                [\n",
       "                    \"35\",\n",
       "                    5\n",
       "                ],\n",
       "                [\n",
       "                    \"36\",\n",
       "                    5\n",
       "                ],\n",
       "                [\n",
       "                    \"37\",\n",
       "                    4\n",
       "                ],\n",
       "                [\n",
       "                    \"38\",\n",
       "                    4\n",
       "                ],\n",
       "                [\n",
       "                    \"39\",\n",
       "                    4\n",
       "                ],\n",
       "                [\n",
       "                    \"40\",\n",
       "                    4\n",
       "                ],\n",
       "                [\n",
       "                    \"41\",\n",
       "                    4\n",
       "                ],\n",
       "                [\n",
       "                    \"42\",\n",
       "                    3\n",
       "                ],\n",
       "                [\n",
       "                    \"43\",\n",
       "                    3\n",
       "                ],\n",
       "                [\n",
       "                    \"44\",\n",
       "                    3\n",
       "                ],\n",
       "                [\n",
       "                    \"45\",\n",
       "                    3\n",
       "                ],\n",
       "                [\n",
       "                    \"46\",\n",
       "                    3\n",
       "                ],\n",
       "                [\n",
       "                    \"47\",\n",
       "                    3\n",
       "                ],\n",
       "                [\n",
       "                    \"48\",\n",
       "                    3\n",
       "                ],\n",
       "                [\n",
       "                    \"49\",\n",
       "                    3\n",
       "                ],\n",
       "                [\n",
       "                    \"50\",\n",
       "                    3\n",
       "                ],\n",
       "                [\n",
       "                    \"51\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"52\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"53\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"54\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"55\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"56\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"57\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"58\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"59\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"60\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"61\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"62\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"63\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"64\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"65\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"66\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"67\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"68\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"69\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"70\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"71\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"72\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"73\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"74\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"75\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"76\",\n",
       "                    2\n",
       "                ],\n",
       "                [\n",
       "                    \"77\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"78\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"79\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"80\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"81\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"82\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"83\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"84\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"85\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"86\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"87\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"88\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"89\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"90\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"91\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"92\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"93\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"94\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"95\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"96\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"97\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"98\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"99\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"100\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"101\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"102\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"103\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"104\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"105\",\n",
       "                    1\n",
       "                ],\n",
       "                [\n",
       "                    \"106\",\n",
       "                    1\n",
       "                ]\n",
       "            ],\n",
       "            \"hoverAnimation\": true,\n",
       "            \"label\": {\n",
       "                \"show\": false,\n",
       "                \"margin\": 8,\n",
       "                \"valueAnimation\": false\n",
       "            },\n",
       "            \"logBase\": 10,\n",
       "            \"seriesLayoutBy\": \"column\",\n",
       "            \"lineStyle\": {\n",
       "                \"show\": true,\n",
       "                \"width\": 2,\n",
       "                \"opacity\": 1,\n",
       "                \"curveness\": 0,\n",
       "                \"type\": \"solid\",\n",
       "                \"color\": \"#3498db\"\n",
       "            },\n",
       "            \"areaStyle\": {\n",
       "                \"opacity\": 0\n",
       "            },\n",
       "            \"itemStyle\": {\n",
       "                \"color\": \"#3498db\"\n",
       "            },\n",
       "            \"zlevel\": 0,\n",
       "            \"z\": 0\n",
       "        }\n",
       "    ],\n",
       "    \"legend\": [\n",
       "        {\n",
       "            \"data\": [\n",
       "                \"Call Count\"\n",
       "            ],\n",
       "            \"selected\": {},\n",
       "            \"show\": true,\n",
       "            \"padding\": 5,\n",
       "            \"itemGap\": 10,\n",
       "            \"itemWidth\": 25,\n",
       "            \"itemHeight\": 14,\n",
       "            \"backgroundColor\": \"transparent\",\n",
       "            \"borderColor\": \"#ccc\",\n",
       "            \"borderRadius\": 0,\n",
       "            \"pageButtonItemGap\": 5,\n",
       "            \"pageButtonPosition\": \"end\",\n",
       "            \"pageFormatter\": \"{current}/{total}\",\n",
       "            \"pageIconColor\": \"#2f4554\",\n",
       "            \"pageIconInactiveColor\": \"#aaa\",\n",
       "            \"pageIconSize\": 15,\n",
       "            \"animationDurationUpdate\": 800,\n",
       "            \"selector\": false,\n",
       "            \"selectorPosition\": \"auto\",\n",
       "            \"selectorItemGap\": 7,\n",
       "            \"selectorButtonGap\": 10\n",
       "        }\n",
       "    ],\n",
       "    \"tooltip\": {\n",
       "        \"show\": true,\n",
       "        \"trigger\": \"axis\",\n",
       "        \"triggerOn\": \"mousemove|click\",\n",
       "        \"axisPointer\": {\n",
       "            \"type\": \"line\"\n",
       "        },\n",
       "        \"showContent\": true,\n",
       "        \"alwaysShowContent\": false,\n",
       "        \"showDelay\": 0,\n",
       "        \"hideDelay\": 100,\n",
       "        \"enterable\": false,\n",
       "        \"confine\": false,\n",
       "        \"appendToBody\": false,\n",
       "        \"transitionDuration\": 0.4,\n",
       "        \"textStyle\": {\n",
       "            \"fontSize\": 14\n",
       "        },\n",
       "        \"borderWidth\": 0,\n",
       "        \"padding\": 5,\n",
       "        \"order\": \"seriesAsc\"\n",
       "    },\n",
       "    \"xAxis\": [\n",
       "        {\n",
       "            \"type\": \"log\",\n",
       "            \"name\": \"Tool Rank\",\n",
       "            \"show\": true,\n",
       "            \"scale\": false,\n",
       "            \"nameLocation\": \"end\",\n",
       "            \"nameGap\": 15,\n",
       "            \"gridIndex\": 0,\n",
       "            \"axisLabel\": {\n",
       "                \"show\": true,\n",
       "                \"margin\": 8,\n",
       "                \"formatter\": \"{value}\",\n",
       "                \"valueAnimation\": false\n",
       "            },\n",
       "            \"inverse\": false,\n",
       "            \"offset\": 0,\n",
       "            \"splitNumber\": 5,\n",
       "            \"minInterval\": 0,\n",
       "            \"splitLine\": {\n",
       "                \"show\": true,\n",
       "                \"lineStyle\": {\n",
       "                    \"show\": true,\n",
       "                    \"width\": 1,\n",
       "                    \"opacity\": 1,\n",
       "                    \"curveness\": 0,\n",
       "                    \"type\": \"solid\"\n",
       "                }\n",
       "            },\n",
       "            \"animation\": true,\n",
       "            \"animationThreshold\": 2000,\n",
       "            \"animationDuration\": 1000,\n",
       "            \"animationEasing\": \"cubicOut\",\n",
       "            \"animationDelay\": 0,\n",
       "            \"animationDurationUpdate\": 300,\n",
       "            \"animationEasingUpdate\": \"cubicOut\",\n",
       "            \"animationDelayUpdate\": 0,\n",
       "            \"data\": [\n",
       "                \"1\",\n",
       "                \"2\",\n",
       "                \"3\",\n",
       "                \"4\",\n",
       "                \"5\",\n",
       "                \"6\",\n",
       "                \"7\",\n",
       "                \"8\",\n",
       "                \"9\",\n",
       "                \"10\",\n",
       "                \"11\",\n",
       "                \"12\",\n",
       "                \"13\",\n",
       "                \"14\",\n",
       "                \"15\",\n",
       "                \"16\",\n",
       "                \"17\",\n",
       "                \"18\",\n",
       "                \"19\",\n",
       "                \"20\",\n",
       "                \"21\",\n",
       "                \"22\",\n",
       "                \"23\",\n",
       "                \"24\",\n",
       "                \"25\",\n",
       "                \"26\",\n",
       "                \"27\",\n",
       "                \"28\",\n",
       "                \"29\",\n",
       "                \"30\",\n",
       "                \"31\",\n",
       "                \"32\",\n",
       "                \"33\",\n",
       "                \"34\",\n",
       "                \"35\",\n",
       "                \"36\",\n",
       "                \"37\",\n",
       "                \"38\",\n",
       "                \"39\",\n",
       "                \"40\",\n",
       "                \"41\",\n",
       "                \"42\",\n",
       "                \"43\",\n",
       "                \"44\",\n",
       "                \"45\",\n",
       "                \"46\",\n",
       "                \"47\",\n",
       "                \"48\",\n",
       "                \"49\",\n",
       "                \"50\",\n",
       "                \"51\",\n",
       "                \"52\",\n",
       "                \"53\",\n",
       "                \"54\",\n",
       "                \"55\",\n",
       "                \"56\",\n",
       "                \"57\",\n",
       "                \"58\",\n",
       "                \"59\",\n",
       "                \"60\",\n",
       "                \"61\",\n",
       "                \"62\",\n",
       "                \"63\",\n",
       "                \"64\",\n",
       "                \"65\",\n",
       "                \"66\",\n",
       "                \"67\",\n",
       "                \"68\",\n",
       "                \"69\",\n",
       "                \"70\",\n",
       "                \"71\",\n",
       "                \"72\",\n",
       "                \"73\",\n",
       "                \"74\",\n",
       "                \"75\",\n",
       "                \"76\",\n",
       "                \"77\",\n",
       "                \"78\",\n",
       "                \"79\",\n",
       "                \"80\",\n",
       "                \"81\",\n",
       "                \"82\",\n",
       "                \"83\",\n",
       "                \"84\",\n",
       "                \"85\",\n",
       "                \"86\",\n",
       "                \"87\",\n",
       "                \"88\",\n",
       "                \"89\",\n",
       "                \"90\",\n",
       "                \"91\",\n",
       "                \"92\",\n",
       "                \"93\",\n",
       "                \"94\",\n",
       "                \"95\",\n",
       "                \"96\",\n",
       "                \"97\",\n",
       "                \"98\",\n",
       "                \"99\",\n",
       "                \"100\",\n",
       "                \"101\",\n",
       "                \"102\",\n",
       "                \"103\",\n",
       "                \"104\",\n",
       "                \"105\",\n",
       "                \"106\"\n",
       "            ]\n",
       "        }\n",
       "    ],\n",
       "    \"yAxis\": [\n",
       "        {\n",
       "            \"type\": \"log\",\n",
       "            \"name\": \"Call Count (Log Scale)\",\n",
       "            \"show\": true,\n",
       "            \"scale\": false,\n",
       "            \"nameLocation\": \"end\",\n",
       "            \"nameGap\": 15,\n",
       "            \"gridIndex\": 0,\n",
       "            \"axisLabel\": {\n",
       "                \"show\": true,\n",
       "                \"margin\": 8,\n",
       "                \"formatter\": \"{value}\",\n",
       "                \"valueAnimation\": false\n",
       "            },\n",
       "            \"inverse\": false,\n",
       "            \"offset\": 0,\n",
       "            \"splitNumber\": 5,\n",
       "            \"minInterval\": 0,\n",
       "            \"splitLine\": {\n",
       "                \"show\": true,\n",
       "                \"lineStyle\": {\n",
       "                    \"show\": true,\n",
       "                    \"width\": 1,\n",
       "                    \"opacity\": 1,\n",
       "                    \"curveness\": 0,\n",
       "                    \"type\": \"solid\"\n",
       "                }\n",
       "            },\n",
       "            \"animation\": true,\n",
       "            \"animationThreshold\": 2000,\n",
       "            \"animationDuration\": 1000,\n",
       "            \"animationEasing\": \"cubicOut\",\n",
       "            \"animationDelay\": 0,\n",
       "            \"animationDurationUpdate\": 300,\n",
       "            \"animationEasingUpdate\": \"cubicOut\",\n",
       "            \"animationDelayUpdate\": 0\n",
       "        }\n",
       "    ],\n",
       "    \"title\": [\n",
       "        {\n",
       "            \"show\": true,\n",
       "            \"text\": \"Tool Call Distribution\",\n",
       "            \"target\": \"blank\",\n",
       "            \"subtarget\": \"blank\",\n",
       "            \"padding\": 5,\n",
       "            \"itemGap\": 10,\n",
       "            \"textAlign\": \"auto\",\n",
       "            \"textVerticalAlign\": \"auto\",\n",
       "            \"triggerEvent\": false\n",
       "        }\n",
       "    ],\n",
       "    \"dataZoom\": [\n",
       "        {\n",
       "            \"show\": true,\n",
       "            \"type\": \"slider\",\n",
       "            \"showDetail\": true,\n",
       "            \"showDataShadow\": true,\n",
       "            \"realtime\": true,\n",
       "            \"start\": 20,\n",
       "            \"end\": 80,\n",
       "            \"orient\": \"horizontal\",\n",
       "            \"zoomLock\": false,\n",
       "            \"filterMode\": \"filter\"\n",
       "        }\n",
       "    ]\n",
       "};\n",
       "                chart_22c006e9322c4a138bd76ffec17ec39f.setOption(option_22c006e9322c4a138bd76ffec17ec39f);\n",
       "        });\n",
       "    </script>\n"
      ],
      "text/plain": [
       "<pyecharts.render.display.HTML at 0x1057ec190>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Prepare the data\n",
    "tools = list(func_cnt.keys())\n",
    "counts = list(func_cnt.values())\n",
    "\n",
    "# Create a long-tail distribution plot (with logarithmic scale)\n",
    "line_chart = (\n",
    "    Line(init_opts=opts.InitOpts(theme=ThemeType.LIGHT, width=\"800px\", height=\"400px\"))\n",
    "    .add_xaxis([str(i) for i in range(1, len(counts) + 1)])\n",
    "    .add_yaxis(\n",
    "        \"Call Count\",\n",
    "        counts,\n",
    "        is_smooth=True,\n",
    "        symbol=\"circle\",\n",
    "        symbol_size=4,\n",
    "        linestyle_opts=opts.LineStyleOpts(width=2, color=\"#3498db\"),\n",
    "        itemstyle_opts=opts.ItemStyleOpts(color=\"#3498db\"),\n",
    "        label_opts=opts.LabelOpts(is_show=False),\n",
    "    )\n",
    "    .set_global_opts(\n",
    "        title_opts=opts.TitleOpts(title=\"Tool Call Distribution\"),\n",
    "        xaxis_opts=opts.AxisOpts(\n",
    "            name=\"Tool Rank\",\n",
    "            type_=\"log\",\n",
    "            axislabel_opts=opts.LabelOpts(formatter=\"{value}\"),\n",
    "        ),\n",
    "        yaxis_opts=opts.AxisOpts(\n",
    "            name=\"Call Count (Log Scale)\",\n",
    "            type_=\"log\",\n",
    "            axislabel_opts=opts.LabelOpts(formatter=\"{value}\"),\n",
    "        ),\n",
    "        tooltip_opts=opts.TooltipOpts(trigger=\"axis\"),\n",
    "        datazoom_opts=[opts.DataZoomOpts()],\n",
    "    )\n",
    ")\n",
    "\n",
    "# Render the long-tail distribution plot\n",
    "line_chart.render_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<script>\n",
       "    require.config({\n",
       "        paths: {\n",
       "            'echarts':'https://assets.pyecharts.org/assets/v5/echarts.min'\n",
       "        }\n",
       "    });\n",
       "</script>\n",
       "\n",
       "        <div id=\"5cbb046c78e1409ab49b3933205771f8\" style=\"width:800px; height:400px;\"></div>\n",
       "\n",
       "<script>\n",
       "        require(['echarts'], function(echarts) {\n",
       "                var chart_5cbb046c78e1409ab49b3933205771f8 = echarts.init(\n",
       "                    document.getElementById('5cbb046c78e1409ab49b3933205771f8'), 'light', {renderer: 'canvas'});\n",
       "                var option_5cbb046c78e1409ab49b3933205771f8 = {\n",
       "    \"animation\": true,\n",
       "    \"animationThreshold\": 2000,\n",
       "    \"animationDuration\": 1000,\n",
       "    \"animationEasing\": \"cubicOut\",\n",
       "    \"animationDelay\": 0,\n",
       "    \"animationDurationUpdate\": 300,\n",
       "    \"animationEasingUpdate\": \"cubicOut\",\n",
       "    \"animationDelayUpdate\": 0,\n",
       "    \"aria\": {\n",
       "        \"enabled\": false\n",
       "    },\n",
       "    \"series\": [\n",
       "        {\n",
       "            \"type\": \"bar\",\n",
       "            \"name\": \"Call Count\",\n",
       "            \"legendHoverLink\": true,\n",
       "            \"data\": [\n",
       "                1678,\n",
       "                48,\n",
       "                44,\n",
       "                34,\n",
       "                33,\n",
       "                32,\n",
       "                31,\n",
       "                28,\n",
       "                28,\n",
       "                24,\n",
       "                22,\n",
       "                22,\n",
       "                19,\n",
       "                18,\n",
       "                17,\n",
       "                17,\n",
       "                16,\n",
       "                12,\n",
       "                12,\n",
       "                11\n",
       "            ],\n",
       "            \"realtimeSort\": false,\n",
       "            \"showBackground\": false,\n",
       "            \"stackStrategy\": \"samesign\",\n",
       "            \"cursor\": \"pointer\",\n",
       "            \"barMinHeight\": 0,\n",
       "            \"barCategoryGap\": \"20%\",\n",
       "            \"barGap\": \"30%\",\n",
       "            \"large\": false,\n",
       "            \"largeThreshold\": 400,\n",
       "            \"seriesLayoutBy\": \"column\",\n",
       "            \"datasetIndex\": 0,\n",
       "            \"clip\": true,\n",
       "            \"zlevel\": 0,\n",
       "            \"z\": 2,\n",
       "            \"label\": {\n",
       "                \"show\": true,\n",
       "                \"position\": \"top\",\n",
       "                \"margin\": 8,\n",
       "                \"valueAnimation\": false\n",
       "            },\n",
       "            \"itemStyle\": {\n",
       "                \"color\": \"#3498db\",\n",
       "                \"borderColor\": \"#3498db\",\n",
       "                \"borderWidth\": 1\n",
       "            }\n",
       "        }\n",
       "    ],\n",
       "    \"legend\": [\n",
       "        {\n",
       "            \"data\": [\n",
       "                \"Call Count\"\n",
       "            ],\n",
       "            \"selected\": {},\n",
       "            \"show\": true,\n",
       "            \"padding\": 5,\n",
       "            \"itemGap\": 10,\n",
       "            \"itemWidth\": 25,\n",
       "            \"itemHeight\": 14,\n",
       "            \"backgroundColor\": \"transparent\",\n",
       "            \"borderColor\": \"#ccc\",\n",
       "            \"borderRadius\": 0,\n",
       "            \"pageButtonItemGap\": 5,\n",
       "            \"pageButtonPosition\": \"end\",\n",
       "            \"pageFormatter\": \"{current}/{total}\",\n",
       "            \"pageIconColor\": \"#2f4554\",\n",
       "            \"pageIconInactiveColor\": \"#aaa\",\n",
       "            \"pageIconSize\": 15,\n",
       "            \"animationDurationUpdate\": 800,\n",
       "            \"selector\": false,\n",
       "            \"selectorPosition\": \"auto\",\n",
       "            \"selectorItemGap\": 7,\n",
       "            \"selectorButtonGap\": 10\n",
       "        }\n",
       "    ],\n",
       "    \"tooltip\": {\n",
       "        \"show\": true,\n",
       "        \"trigger\": \"axis\",\n",
       "        \"triggerOn\": \"mousemove|click\",\n",
       "        \"axisPointer\": {\n",
       "            \"type\": \"shadow\"\n",
       "        },\n",
       "        \"showContent\": true,\n",
       "        \"alwaysShowContent\": false,\n",
       "        \"showDelay\": 0,\n",
       "        \"hideDelay\": 100,\n",
       "        \"enterable\": false,\n",
       "        \"confine\": false,\n",
       "        \"appendToBody\": false,\n",
       "        \"transitionDuration\": 0.4,\n",
       "        \"formatter\": \"{b}: {c} calls\",\n",
       "        \"textStyle\": {\n",
       "            \"fontSize\": 14\n",
       "        },\n",
       "        \"borderWidth\": 0,\n",
       "        \"padding\": 5,\n",
       "        \"order\": \"seriesAsc\"\n",
       "    },\n",
       "    \"xAxis\": [\n",
       "        {\n",
       "            \"name\": \"Tool Name\",\n",
       "            \"show\": true,\n",
       "            \"scale\": false,\n",
       "            \"nameLocation\": \"end\",\n",
       "            \"nameGap\": 15,\n",
       "            \"gridIndex\": 0,\n",
       "            \"axisLabel\": {\n",
       "                \"show\": true,\n",
       "                \"rotate\": 45,\n",
       "                \"margin\": 8,\n",
       "                \"valueAnimation\": false\n",
       "            },\n",
       "            \"inverse\": false,\n",
       "            \"offset\": 0,\n",
       "            \"splitNumber\": 5,\n",
       "            \"minInterval\": 0,\n",
       "            \"splitLine\": {\n",
       "                \"show\": true,\n",
       "                \"lineStyle\": {\n",
       "                    \"show\": true,\n",
       "                    \"width\": 1,\n",
       "                    \"opacity\": 1,\n",
       "                    \"curveness\": 0,\n",
       "                    \"type\": \"solid\"\n",
       "                }\n",
       "            },\n",
       "            \"animation\": true,\n",
       "            \"animationThreshold\": 2000,\n",
       "            \"animationDuration\": 1000,\n",
       "            \"animationEasing\": \"cubicOut\",\n",
       "            \"animationDelay\": 0,\n",
       "            \"animationDurationUpdate\": 300,\n",
       "            \"animationEasingUpdate\": \"cubicOut\",\n",
       "            \"animationDelayUpdate\": 0,\n",
       "            \"data\": [\n",
       "                \"<|NO_FUNCTION_C...\",\n",
       "                \"calculate_dista...\",\n",
       "                \"convert_currenc...\",\n",
       "                \"calculate_tip\",\n",
       "                \"calculate_area\",\n",
       "                \"calculate_disco...\",\n",
       "                \"get_stock_price\",\n",
       "                \"calculate_age\",\n",
       "                \"get_movie_detai...\",\n",
       "                \"calculate_bmi\",\n",
       "                \"search_books\",\n",
       "                \"search_recipes\",\n",
       "                \"calculate_loan_...\",\n",
       "                \"generate_random...\",\n",
       "                \"send_email\",\n",
       "                \"generate_passwo...\",\n",
       "                \"generate_qr_cod...\",\n",
       "                \"generate_random...\",\n",
       "                \"create_calendar...\",\n",
       "                \"search_movie\"\n",
       "            ]\n",
       "        }\n",
       "    ],\n",
       "    \"yAxis\": [\n",
       "        {\n",
       "            \"name\": \"Call Count\",\n",
       "            \"show\": true,\n",
       "            \"scale\": false,\n",
       "            \"nameLocation\": \"end\",\n",
       "            \"nameGap\": 15,\n",
       "            \"gridIndex\": 0,\n",
       "            \"inverse\": false,\n",
       "            \"offset\": 0,\n",
       "            \"splitNumber\": 5,\n",
       "            \"minInterval\": 0,\n",
       "            \"splitLine\": {\n",
       "                \"show\": true,\n",
       "                \"lineStyle\": {\n",
       "                    \"show\": true,\n",
       "                    \"width\": 1,\n",
       "                    \"opacity\": 1,\n",
       "                    \"curveness\": 0,\n",
       "                    \"type\": \"solid\"\n",
       "                }\n",
       "            },\n",
       "            \"animation\": true,\n",
       "            \"animationThreshold\": 2000,\n",
       "            \"animationDuration\": 1000,\n",
       "            \"animationEasing\": \"cubicOut\",\n",
       "            \"animationDelay\": 0,\n",
       "            \"animationDurationUpdate\": 300,\n",
       "            \"animationEasingUpdate\": \"cubicOut\",\n",
       "            \"animationDelayUpdate\": 0\n",
       "        }\n",
       "    ],\n",
       "    \"title\": [\n",
       "        {\n",
       "            \"show\": true,\n",
       "            \"text\": \"Top 20 Most Used Tools\",\n",
       "            \"target\": \"blank\",\n",
       "            \"subtarget\": \"blank\",\n",
       "            \"padding\": 5,\n",
       "            \"itemGap\": 10,\n",
       "            \"textAlign\": \"auto\",\n",
       "            \"textVerticalAlign\": \"auto\",\n",
       "            \"triggerEvent\": false\n",
       "        }\n",
       "    ]\n",
       "};\n",
       "                chart_5cbb046c78e1409ab49b3933205771f8.setOption(option_5cbb046c78e1409ab49b3933205771f8);\n",
       "        });\n",
       "    </script>\n"
      ],
      "text/plain": [
       "<pyecharts.render.display.HTML at 0x105422dc0>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a bar chart for the top 20 most used tools\n",
    "top_20_tools = tools[:20]\n",
    "top_20_counts = counts[:20]\n",
    "\n",
    "bar_chart = (\n",
    "    Bar(init_opts=opts.InitOpts(theme=ThemeType.LIGHT, width=\"800px\", height=\"400px\"))\n",
    "    .add_xaxis([tool[:15] + \"...\" if len(tool) > 15 else tool for tool in top_20_tools])\n",
    "    .add_yaxis(\n",
    "        \"Call Count\",\n",
    "        top_20_counts,\n",
    "        itemstyle_opts=opts.ItemStyleOpts(\n",
    "            color=\"#3498db\",\n",
    "            border_color=\"#3498db\",\n",
    "            border_width=1,\n",
    "        ),\n",
    "        label_opts=opts.LabelOpts(is_show=True, position=\"top\"),\n",
    "    )\n",
    "    .set_global_opts(\n",
    "        title_opts=opts.TitleOpts(title=\"Top 20 Most Used Tools\"),\n",
    "        xaxis_opts=opts.AxisOpts(\n",
    "            name=\"Tool Name\",\n",
    "            axislabel_opts=opts.LabelOpts(rotate=45),\n",
    "        ),\n",
    "        yaxis_opts=opts.AxisOpts(name=\"Call Count\"),\n",
    "        tooltip_opts=opts.TooltipOpts(trigger=\"axis\", axis_pointer_type=\"shadow\", formatter=\"{b}: {c} calls\"),\n",
    "    )\n",
    ")\n",
    "\n",
    "# Render the bar chart\n",
    "bar_chart.render_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of tools: 106\n",
      "Total number of calls: 2382\n",
      "Average calls per tool: 22.47\n",
      "Most frequently called tool: <|NO_FUNCTION_CALL|> (1678 calls)\n",
      "Number of tools called only once: 30\n",
      "Number of tools called 5 or more times: 36\n"
     ]
    }
   ],
   "source": [
    "# Print statistical information\n",
    "total_tools = len(func_cnt)\n",
    "total_calls = sum(func_cnt.values())\n",
    "print(f\"Total number of tools: {total_tools}\")\n",
    "print(f\"Total number of calls: {total_calls}\")\n",
    "print(f\"Average calls per tool: {total_calls/total_tools:.2f}\")\n",
    "print(f\"Most frequently called tool: {tools[0]} ({counts[0]} calls)\")\n",
    "print(f\"Number of tools called only once: {sum(1 for c in counts if c == 1)}\")\n",
    "print(f\"Number of tools called 5 or more times: {sum(1 for c in counts if c >= 5)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the system prompt. After some simple experiments, we will use the system prompt that performed best in our tests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "system_prompt_template = Template(\n",
    "    \"\"\"You are an intelligent assistant that can find the most suitable tool to complete a task based on user needs. Please refer to the following list of tools to complete the task. Note:\n",
    "1. Only return the tool name, do not return any other content.\n",
    "2. Do not execute the task itself, only return the tool name. For example, if the user needs to convert Celsius to Fahrenheit, you only need to return the string 'convert_temperature'.\n",
    "3. If you think no tool needs to be called, please return the string '<|NO_FUNCTION_CALL|>'. For example, if the user is just asking for general information, or has explicitly requested not to use a tool.\n",
    "4. You can only call tools from the list; do not make up tools.\n",
    "\n",
    "Available tools:\n",
    "${tools_prompt}\"\"\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "tools_prompt = \"\"\n",
    "for tool_name, tool_desc in func_name_with_desc.items():\n",
    "    tools_prompt += f\"Tool Name: {tool_name}\\n\"\n",
    "    tools_prompt += f\"Tool Description: {tool_desc}\\n\"\n",
    "    tools_prompt += \"\\n\"\n",
    "tools_prompt = tools_prompt.strip()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Evaluate the Base Model's Capabilities\n",
    "Define the evaluation metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_multiclass_metrics(data):\n",
    "    \"\"\"\n",
    "    Calculates the accuracy, precision, recall, and F1 score for a multi-class classification task.\n",
    "\n",
    "    Args:\n",
    "        data: A list where each element is a dictionary in the format {\"ground_truth\": \"\", \"prediction\": \"\"}.\n",
    "\n",
    "    Returns:\n",
    "        A dictionary containing overall metrics and per-class metrics.\n",
    "    \"\"\"\n",
    "    if not data:\n",
    "        return {}\n",
    "\n",
    "    # Extract all ground truth and predicted labels\n",
    "    y_true = [item[\"ground_truth\"] for item in data]\n",
    "    y_pred = [item[\"prediction\"] for item in data]\n",
    "\n",
    "    # Only consider classes present in the ground truth labels\n",
    "    true_classes = set(y_true)\n",
    "\n",
    "    # Calculate overall accuracy\n",
    "    correct = sum(1 for gt, pred in zip(y_true, y_pred) if gt == pred)\n",
    "    total = len(data)\n",
    "    accuracy = correct / total if total > 0 else 0\n",
    "\n",
    "    # Calculate metrics for each true class\n",
    "    class_metrics = {}\n",
    "    precision_list = []\n",
    "    recall_list = []\n",
    "    f1_list = []\n",
    "    support_list = []\n",
    "\n",
    "    for cls in true_classes:\n",
    "        # Calculate TP, FP, FN\n",
    "        tp = sum(1 for gt, pred in zip(y_true, y_pred) if gt == cls and pred == cls)\n",
    "        fp = sum(1 for gt, pred in zip(y_true, y_pred) if gt != cls and pred == cls)\n",
    "        fn = sum(1 for gt, pred in zip(y_true, y_pred) if gt == cls and pred != cls)\n",
    "\n",
    "        # Calculate precision, recall, F1 (maintaining original precision)\n",
    "        precision = tp / (tp + fp) if (tp + fp) > 0 else 0\n",
    "        recall = tp / (tp + fn) if (tp + fn) > 0 else 0\n",
    "        f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0\n",
    "        support = sum(1 for gt in y_true if gt == cls)\n",
    "\n",
    "        class_metrics[cls] = {\n",
    "            \"precision\": round(precision * 100, 3),\n",
    "            \"recall\": round(recall * 100, 3),\n",
    "            \"f1_score\": round(f1 * 100, 3),\n",
    "            \"support\": support,\n",
    "        }\n",
    "\n",
    "        # Collect raw values for calculating averages\n",
    "        precision_list.append(precision)\n",
    "        recall_list.append(recall)\n",
    "        f1_list.append(f1)\n",
    "        support_list.append(support)\n",
    "\n",
    "    # Calculate macro average - using raw precision values\n",
    "    macro_precision = sum(precision_list) / len(precision_list) if precision_list else 0\n",
    "    macro_recall = sum(recall_list) / len(recall_list) if recall_list else 0\n",
    "    macro_f1 = sum(f1_list) / len(f1_list) if f1_list else 0\n",
    "\n",
    "    # Calculate micro average\n",
    "    total_tp = sum(sum(1 for gt, pred in zip(y_true, y_pred) if gt == cls and pred == cls) for cls in true_classes)\n",
    "    total_fp = sum(sum(1 for gt, pred in zip(y_true, y_pred) if gt != cls and pred == cls) for cls in true_classes)\n",
    "    total_fn = sum(sum(1 for gt, pred in zip(y_true, y_pred) if gt == cls and pred != cls) for cls in true_classes)\n",
    "\n",
    "    micro_precision = total_tp / (total_tp + total_fp) if (total_tp + total_fp) > 0 else 0\n",
    "    micro_recall = total_tp / (total_tp + total_fn) if (total_tp + total_fn) > 0 else 0\n",
    "    micro_f1 = (\n",
    "        2 * micro_precision * micro_recall / (micro_precision + micro_recall)\n",
    "        if (micro_precision + micro_recall) > 0\n",
    "        else 0\n",
    "    )\n",
    "\n",
    "    # Calculate weighted average - using raw precision values\n",
    "    total_support = sum(support_list)\n",
    "    weighted_precision = (\n",
    "        sum(p * s for p, s in zip(precision_list, support_list)) / total_support if total_support > 0 else 0\n",
    "    )\n",
    "    weighted_recall = sum(r * s for r, s in zip(recall_list, support_list)) / total_support if total_support > 0 else 0\n",
    "    weighted_f1 = sum(f * s for f, s in zip(f1_list, support_list)) / total_support if total_support > 0 else 0\n",
    "\n",
    "    return {\n",
    "        \"accuracy\": f\"{round(accuracy * 100, 3)}%\",\n",
    "        \"macro_avg\": {\n",
    "            \"precision\": f\"{round(macro_precision * 100, 3)}%\",\n",
    "            \"recall\": f\"{round(macro_recall * 100, 3)}%\",\n",
    "            \"f1_score\": f\"{round(macro_f1 * 100, 3)}%\",\n",
    "        },\n",
    "        \"micro_avg\": {\n",
    "            \"precision\": f\"{round(micro_precision * 100, 3)}%\",\n",
    "            \"recall\": f\"{round(micro_recall * 100, 3)}%\",\n",
    "            \"f1_score\": f\"{round(micro_f1 * 100, 3)}%\",\n",
    "        },\n",
    "        \"weighted_avg\": {\n",
    "            \"precision\": f\"{round(weighted_precision * 100, 3)}%\",\n",
    "            \"recall\": f\"{round(weighted_recall * 100, 3)}%\",\n",
    "            \"f1_score\": f\"{round(weighted_f1 * 100, 3)}%\",\n",
    "        },\n",
    "        \"per_class\": class_metrics,\n",
    "        \"total_samples\": total,\n",
    "        \"correct_predictions\": correct,\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Construct the test set. Based on the analysis above, the distribution of functions in the original dataset is highly imbalanced. We will focus on functions that have at least 5 samples, and randomly select 2 instances for each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(history_path, \"r\") as fin:\n",
    "    all_histories = [json.loads(line) for line in fin]\n",
    "\n",
    "# Find frequently called tools\n",
    "frequent_funcs = {func_name: count for func_name, count in func_cnt.items() if count >= 5}\n",
    "\n",
    "# Find the dialogue indices corresponding to the frequently called tools\n",
    "history_idx_called_frequent_func = defaultdict(list)\n",
    "for idx, history in enumerate(all_histories):\n",
    "    func_name = history[-1].get(\"content\", \"\")\n",
    "    if func_name in frequent_funcs:\n",
    "        history_idx_called_frequent_func[func_name].append(idx)\n",
    "\n",
    "random.seed(RANDOM_SEED)\n",
    "test_indices = set()\n",
    "for tool, history_indices in history_idx_called_frequent_func.items():\n",
    "    selected = random.sample(history_indices, 2)\n",
    "    test_indices.update(selected)\n",
    "\n",
    "# Save the test set: one version without ground truth for inference, and another with ground truth for evaluation.\n",
    "testset_no_gt_path = \"../data/sft_testset_no_gt.jsonl\"\n",
    "testset_has_gt_path = \"../data/sft_testset_has_gt.jsonl\"\n",
    "with open(testset_no_gt_path, \"w\") as no_gt, open(testset_has_gt_path, \"w\") as has_gt:\n",
    "    for idx in test_indices:\n",
    "        testset_history = all_histories[idx]\n",
    "        ground_truth = testset_history.pop().get(\"content\", \"\")\n",
    "        no_gt.write(\n",
    "            json.dumps(\n",
    "                history_to_test(\n",
    "                    history=testset_history, system_prompt=system_prompt_template.substitute(tools_prompt=tools_prompt)\n",
    "                ),\n",
    "                ensure_ascii=False,\n",
    "            )\n",
    "            + \"\\n\"\n",
    "        )\n",
    "        has_gt.write(json.dumps({\"history\": testset_history, \"ground_truth\": ground_truth}, ensure_ascii=False) + \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run inference on the test set using the base model. The WebUI does not currently support inference on test sets, so we will use an inference script from the ERNIEKit codebase.\n",
    "- **Script**: `examples/inference/scripts/infer.sh`\n",
    "- **Set the main hyperparameters**:\n",
    "  - `top_p=0.7`\n",
    "  - `temperature=0.7`\n",
    "  - `max_seq_len=128000` (ERNIE-4.5-0.3B supports a dynamic 128K context length)\n",
    "  - `max_dec_len=64` (our task does not require a long output)\n",
    "  - `weight_quantize_algo`: This line can be deleted if quantization was not used during training.\n",
    "\n",
    "> Inference process omitted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate the evaluation metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get inference results\n",
    "prediction_data_path = \"../data/infer/sft_03b_exp0.jsonl\"\n",
    "\n",
    "# To handle potentially out-of-order results, we'll reorganize and map the inference results to the input information.\n",
    "history2gt = {}\n",
    "with open(testset_has_gt_path, \"r\") as fin:\n",
    "    for line in fin:\n",
    "        data_eval = json.loads(line.strip())\n",
    "        history2gt[json.dumps(data_eval[\"history\"], ensure_ascii=False)] = data_eval[\"ground_truth\"]\n",
    "\n",
    "history2pred = {}\n",
    "with open(prediction_data_path, \"r\") as fin:\n",
    "    for line in fin:\n",
    "        data_pred = json.loads(line.strip())\n",
    "        history2pred[json.dumps(data_pred[1:-1], ensure_ascii=False)] = data_pred[-1][\"content\"]\n",
    "\n",
    "pred2gt = []\n",
    "for history, pred in history2pred.items():\n",
    "    pred2gt.append({\"ground_truth\": history2gt[history], \"prediction\": pred})\n",
    "\n",
    "# Calculate multi-class metrics\n",
    "results = calculate_multiclass_metrics(pred2gt)\n",
    "print(\"Evaluation Results:\")\n",
    "print(f\"Accuracy: {results['accuracy']}\")\n",
    "print(\n",
    "    f\"Macro Avg - Precision: {results['macro_avg']['precision']}, Recall: {results['macro_avg']['recall']}, F1: {results['macro_avg']['f1_score']}\"\n",
    ")\n",
    "print(\n",
    "    f\"Micro Avg - Precision: {results['micro_avg']['precision']}, Recall: {results['micro_avg']['recall']}, F1: {results['micro_avg']['f1_score']}\"\n",
    ")\n",
    "print(\n",
    "    f\"Weighted Avg - Precision: {results['weighted_avg']['precision']}, Recall: {results['weighted_avg']['recall']}, F1: {results['weighted_avg']['f1_score']}\"\n",
    ")\n",
    "\n",
    "\n",
    "# Convert per-class results to a DataFrame for easier analysis\n",
    "per_class_data = []\n",
    "for class_name, metrics in results['per_class'].items():\n",
    "    per_class_data.append(\n",
    "        {\n",
    "            'Class': class_name,\n",
    "            'Precision(%)': str(metrics['precision']).rstrip('%'),\n",
    "            'Recall(%)': str(metrics['recall']).rstrip('%'),\n",
    "            'F1 Score(%)': str(metrics['f1_score']).rstrip('%'),\n",
    "            'Support': metrics['support'],\n",
    "        }\n",
    "    )\n",
    "\n",
    "df_per_class = pd.DataFrame(per_class_data)\n",
    "print(\"\\nDetailed Metrics Per Class:\")\n",
    "df_per_class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4 Making Training Set\n",
    "Evidently, the current model is completely unable to select the appropriate tool functions, making SFT a necessary step. Therefore, we will construct the training and validation sets. Of the data not included in the test set, 10% will be allocated for validation, and the remainder will be used for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainset_path = \"../data/sft_trainset.jsonl\"\n",
    "validset_path = \"../data/sft_validset.jsonl\"\n",
    "\n",
    "# Split the validation set from the non-test data\n",
    "remaining_indices = [idx for idx in range(len(all_histories)) if idx not in test_indices]\n",
    "random.shuffle(remaining_indices)\n",
    "valid_size = int(len(remaining_indices) * 0.1)\n",
    "valid_indices = set(remaining_indices[:valid_size])\n",
    "train_indices = set(remaining_indices[valid_size:])\n",
    "\n",
    "# Save the training set\n",
    "with open(trainset_path, \"w\") as fout:\n",
    "    for idx in train_indices:\n",
    "        history = all_histories[idx]\n",
    "        train_data = history_to_train(\n",
    "            history=history, system_prompt=system_prompt_template.substitute(tools_prompt=tools_prompt)\n",
    "        )\n",
    "        fout.write(json.dumps(train_data, ensure_ascii=False) + \"\\n\")\n",
    "\n",
    "# Save the validation set\n",
    "with open(validset_path, \"w\") as fout:\n",
    "    for idx in valid_indices:\n",
    "        history = all_histories[idx]\n",
    "        valid_data = history_to_train(\n",
    "            history=history, system_prompt=system_prompt_template.substitute(tools_prompt=tools_prompt)\n",
    "        )\n",
    "        fout.write(json.dumps(valid_data, ensure_ascii=False) + \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Training\n",
    "### 4.1 Select the Base Model\n",
    "For ease of demonstration and considering the task's complexity, we'll choose the smaller ERNIE-4.5-0.3B model.\n",
    "\n",
    "### 4.2 Design Training Parameters\n",
    "Designing training parameters is largely based on experience. ERNIEKit allows users to set a few key parameters and automatically recommends the rest. Typically, the key parameters you need to set are:\n",
    "- `num_train_epochs`\n",
    "- `gradient_accumulation_steps`\n",
    "- `max_seq_len=8192`\n",
    "\n",
    "Calculate recommended parameters (see `examples/post-training/sft/README.md` for details):\n",
    "```bash\n",
    "python examples/post-training/sft/estimate_training.py \\\n",
    "    --train_task_config examples/post-training/sft/config/task_sft.json \\\n",
    "    --model_name_or_path baidu/paddle_internal/ERNIE-4.5-0.3B \\\n",
    "    --out_file estimate_training.json \\\n",
    "    --num_train_epochs 1 \\\n",
    "    --max_seq_len 8192 \\\n",
    "    --num_of_gpus 1 \\\n",
    "    --per_device_train_batch_size 1 \\\n",
    "    --tensor_parallel_degree 1 \\\n",
    "    --pipeline_parallel_degree 1 \\\n",
    "    --sharding_parallel_degree 1 \\\n",
    "    --seed 23 \\\n",
    "    --num_samples_each_epoch 6000000\n",
    "```\n",
    "\n",
    "Here, `train_task_config` is a JSON file that records training data information, like this:\n",
    "```json\n",
    "[\n",
    "    {\n",
    "        \"prob\": 1,\n",
    "        \"filepath\": \"data/sft_trainset.jsonl\"\n",
    "    }\n",
    "]\n",
    "```\n",
    "\n",
    "### 4.3 Train with the WebUI\n",
    "1. Launch the WebUI from the ERNIEKit project's root directory: `erniekit webui`\n",
    "2. Configure the model path and export directory.\n",
    "\n",
    "![sft_set_path](../assets/sft_set_path_en.png)\n",
    "\n",
    "3. Set up full-parameter/LoRA fine-tuning, numerical precision, etc.\n",
    "\n",
    "![sft_lora_set](../assets/sft_set_lora_en.png)\n",
    "\n",
    "4. Set the fine-tuning mode (SFT/DPO) and configure training parameters.\n",
    "\n",
    "![sft_train_params](../assets/sft_train_params_en.png)\n",
    "\n",
    "5. Configure the training and validation sets.\n",
    "\n",
    "<div style=\"display: flex; justify-content: space-around;\">\n",
    "    <img src=\"../assets/sft_trainset_en.png\" alt=\"sft_trainset\" style=\"width: 45%;\">\n",
    "    <img src=\"../assets/sft_validset_en.png\" alt=\"sft_validset\" style=\"width: 45%;\">\n",
    "</div>\n",
    "\n",
    "### 4.4 View Training Logs and Loss Curves\n",
    "The training log path is `${your_model_dir}/paddle_dist_log/workerlog.0`. You can also run the following command to view the loss curve:\n",
    "```bash\n",
    "visualdl --logdir ${your_model_dir}/vdl_log --host 0.0.0.0\n",
    "```\n",
    "Here is an example of a loss curve viewed in VisualDL:\n",
    "\n",
    "![sft_visualdl_loss](../assets/sft_visualdl_loss.png)\n",
    "\n",
    "### 4.5 Merge Model Weights (Only for LoRA Fine-Tuning)\n",
    "You can easily merge model weights in the WebUI's \"Evaluation\" mode.\n",
    "\n",
    "![sft_merge_lora](../assets/sft_merge_lora_en.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Performance Evaluation\n",
    "\n",
    "### 5.1 Calculate Evaluation Metrics\n",
    "We have already defined the evaluation criteria and demonstrated how to calculate the metrics. Here, we will not repeat that but will instead present the training configurations and evaluation metrics from several experiments for your reference. We also tested the performance of the ERNIE-4.5-21B-A3B model with the same configuration.\n",
    "\n",
    "- ERNIE-4.5-0.3B  \n",
    "\n",
    "| Experiment No. | Manually Configured Parameters | Corresponding Recommended Parameters | Metrics |\n",
    "| --- | --- | --- | --- |\n",
    "| 0 | Base model without SFT | - | ↑Accuracy: 2.532% <br> ↑Macro Avg - Precision: 2.75%, Recall: 2.5%, F1: 2.083% <br> ↑Micro Avg - Precision: 5.714%, Recall: 2.532%, F1: 3.509% <br> ↑Weighted Avg - Precision: 2.785%, Recall: 2.532%, F1: 2.11% |\n",
    "| 1 | num_train_epochs=1 <br> gradient_accumulation_steps=1 <br> max_seq_len=8192 <br> num_of_gpus=4 <br> tensor_parallel_degree=4 | max_steps=1070 <br> global_batch_size=1 <br> warmup_steps=107 | ↑Accuracy: 87.342% <br> ↑Macro Avg - Precision: 89.583%, Recall: 87.5%, F1: 87.0% <br> ↑Micro Avg - Precision: 90.789%, Recall: 87.342%, F1: 89.032% <br> ↑Weighted Avg - Precision: 89.451%, Recall: 87.342%, F1: 86.835% |\n",
    "\n",
    "- ERNIE-4.5-21B-A3B  \n",
    "\n",
    "| Experiment No. | Manually Configured Parameters | Corresponding Recommended Parameters | Metrics |\n",
    "| ---- | ---- | ---- | ---- |\n",
    "| 0 | Base model without SFT | - | ↑Accuracy: 48.101%<br>↑Macro Avg - Precision: 54.583%, Recall: 48.75%, F1: 49.333%<br>↑Micro Avg - Precision: 52.055%, Recall: 48.101%, F1: 50.0%<br>↑Weighted Avg - Precision: 54.008%, Recall: 48.101%, F1: 48.692% |\n",
    "| 1 | num_train_epochs=1<br>gradient_accumulation_steps=1<br>max_seq_len=8192<br>num_of_gpus=4<br>tensor_parallel_degree=4 | max_steps=1070<br>global_batch_size=1<br>warmup_steps=107 | ↑Accuracy: 89.873%<br>↑Macro Avg - Precision: 87.083%, Recall: 90.0%, F1: 87.333%<br>↑Micro Avg - Precision: 89.873%, Recall: 89.873%, F1: 89.873%<br>↑Weighted Avg - Precision: 86.92%, Recall: 89.873%, F1: 87.173% |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 Analyze the Experimental Results\n",
    "As we can see, the current model has achieved impressive metrics. However, there's still room for exploration and improvement. We have prepared a few data analysis tools to help us find clues for further optimizing the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_histories_by_function_name(function_name, eval_data_path, prediction_data_path):\n",
    "    \"\"\"\n",
    "    Retrieves the corresponding dialogue histories for a given function name.\n",
    "\n",
    "    Args:\n",
    "        function_name: The name of the function to search for.\n",
    "        eval_data_path: The file path for the evaluation data (with ground truth).\n",
    "        prediction_data_path: The file path for the prediction data.\n",
    "\n",
    "    Returns:\n",
    "        A tuple (gt_histories, pred_histories), where each element is a list of\n",
    "        histories that match the ground truth and prediction labels, respectively.\n",
    "    \"\"\"\n",
    "    history2gt = {}\n",
    "    with open(eval_data_path, \"r\") as fin:\n",
    "        for line in fin:\n",
    "            data_eval = json.loads(line.strip())\n",
    "            history2gt[json.dumps(data_eval[\"history\"], ensure_ascii=False)] = data_eval[\"ground_truth\"]\n",
    "\n",
    "    history2pred = {}\n",
    "    with open(prediction_data_path, \"r\") as fin:\n",
    "        for line in fin:\n",
    "            data_pred = json.loads(line.strip())\n",
    "            history2pred[json.dumps(data_pred[1:-1], ensure_ascii=False)] = data_pred[-1][\"content\"]\n",
    "\n",
    "    # Find histories where the ground truth matches the function name\n",
    "    gt_histories = []\n",
    "    for history_str, gt in history2gt.items():\n",
    "        if gt == function_name:\n",
    "            history_data = json.loads(history_str)\n",
    "            history_data.append({\"ground_truth\": function_name})\n",
    "            gt_histories.append(history_data)\n",
    "\n",
    "    # Find histories where the prediction matches the function name\n",
    "    pred_histories = []\n",
    "    for history_str, pred in history2pred.items():\n",
    "        if pred == function_name:\n",
    "            history_data = json.loads(history_str)\n",
    "            history_data.append({\"prediction\": function_name})\n",
    "            pred_histories.append(history_data)\n",
    "\n",
    "    return gt_histories, pred_histories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.3 Catastrophic Forgetting\n",
    "We have observed that the model's performance declines after SFT on the function-calling task, exhibiting a degree of \"catastrophic forgetting.\" For example:\n",
    "\n",
    "> Input: `[{\"role\": \"user\", \"content\": \"Please remember, your name is Pipilu\"}, {\"role\": \"assistant\", \"content\": \"Okay, I am Pipilu\"}, {\"role\": \"user\", \"content\": \"Who are you?\"}]`\n",
    "\n",
    "> ERNIE-4.5-0.3B Base Model Output: I am Pipilu! How can I help you?\n",
    "> SFT Model Output: `get_definition`\n",
    "\n",
    "This is a common phenomenon in large language model SFT. The data for this task is relatively specialized, causing SFT to enhance the model's performance on this specific task while degrading its general capabilities. To maintain the model's general abilities, we can simply adopt a **data mixing strategy** to alleviate this issue. ERNIE provides several general-purpose SFT datasets that can be mixed with downstream task data (like the function-calling data here) for SFT."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6 FAQ\n",
    "\n",
    "### 6.1 What should I do if I can't fine-tune the 32B model even with four 80GB A800 GPUs?\n",
    "\n",
    "You can configure the \"Distributed Parameters\" in the WebUI as follows:\n",
    "\n",
    "![sft_21b_distributed_params_en](../assets/sft_21b_distributed_params_en.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ar",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
